Matching support apparatus, matching support method, and computer-readable recording medium

ABSTRACT

A matching support apparatus includes a first display information generation unit that generates first display information for displaying, on the screen of a display device, a first display region for displaying an image including the face of a person targeted for matching captured using an image capturing apparatus, a second display information generation unit that generates second display information for displaying, on the screen of the display device 30, a second display region for displaying a reference face development image generated based on three-dimensional data of a head serving as a reference, and a user interface display information generation unit that generates first user interface display information for displaying, on the screen of the display device 30, a second user interface to be used in an operation for enabling a user to designate, in the second display region, a feature region of the person targeted for matching visible on the skin surface of the person targeted for matching.

TECHNICAL FIELD

The present invention relates to a matching support apparatus and amatching support method for supporting matching, and further relates toa computer-readable recording medium that includes a program recordedthereon for realizing the apparatus and method.

BACKGROUND ART

Matching apparatuses have been proposed that perform matching using theface image of a targeted person and preregistered face images, andspecify the targeted person based on a matching result.

For example, as a related technology, Patent Document 1 discloses anauthentication system that is able to perform authentication with highaccuracy in the case of authenticating the identity of a person.According to the authentication system of Patent Document 1, a featureregion corresponding to a discrete feature site (mole, scar, wrinkle) isautomatically detected from an image captured of the person targeted forauthentication, a feature amount of the detected feature region isrecognized, and authentication is executed using the recognized featureamount.

LIST OF RELATED ART DOCUMENTS Patent Document

Patent Document 1: Japanese Patent Laid-Open Publication No. 2007-304857

SUMMARY Technical Problems

However, with the authentication system of Patent Document 1,preregistered face images are used, and thus matching may not bepossible in the case where the orientation of the face in the registeredface images is different from the orientation of the face of thetargeted person in the captured image. For example, in the case wherethe face of the targeted person in the captured image is not facingforward, the apparent position of the discrete feature site changes dueto the undulations of the face, and thus a forward-facing face imagemust always be used.

Accordingly, in the case where an image of a face that is not facingforward is used, performing matching using the face image will bedifficult. In view of this, there are calls to also be able to designatea feature region corresponding to a discrete feature site, when using animage of a face that is not facing forward.

An example object of the invention is to provide a matching supportapparatus, a matching support method and a computer-readable recordingmedium with which a feature visible on the skin surface of the face of aperson targeted for matching can be designated, according to theorientation of the face in a captured image.

Solution to the Problems

A matching support apparatus according to an example aspect of theinvention includes:

a first display information generation means for generating firstdisplay information for displaying, on a screen of a display device, afirst display region for displaying an image including a face of aperson targeted for matching captured using an image capturingapparatus;

a second display information generation means for generating seconddisplay information for displaying, on the screen of the display device,a second display region for displaying a reference face developmentimage generated based on three-dimensional data of a head serving as areference; and

a user interface display information generation means for generatingfirst user interface display information for displaying, on the screenof the display device, a first user interface to be used in an operationfor enabling a user to designate, in the second display region, afeature region indicating a facial feature of the person targeted formatching visible on a skin surface of the person targeted for matching.

Also, a matching support method according to an example aspect of theinvention includes:

(a) generating first display information for displaying, on a screen ofa display device, a first display region for displaying an imageincluding a face of a person targeted for matching captured using animage capturing apparatus;

(b) generating second display information for displaying, on the screenof the display device, a second display region for displaying areference face development image generated based on three-dimensionaldata of a head serving as a reference; and

(c) generating first user interface display information for displaying,on the screen of the display device, a first user interface to be usedin an operation for enabling a user to designate, in the second displayregion, a feature region indicating a facial feature of the persontargeted for matching visible on a skin surface of the person targetedfor matching.

Furthermore, a computer-readable recording medium according to anexample aspect of the invention includes a program recorded thereon, theprogram including instruction that cause a computer to carry out:

(a) a step of generating first display information for displaying, on ascreen of a display device, a first display region for displaying animage including a face of a person targeted for matching captured usingan image capturing apparatus;

(b) a step of generating second display information for displaying, onthe screen of the display device, a second display region for displayinga reference face development image generated based on three-dimensionaldata of a head serving as a reference; and

(c) a step of generating first user interface display information fordisplaying, on the screen of the display device, a first user interfaceto be used in an operation for enabling a user to designate, in thesecond display region, a feature region indicating a facial feature ofthe person targeted for matching visible on a skin surface of the persontargeted for matching.

Advantageous Effects

According to the invention as described above, a feature visible on theskin surface of the face of a person targeted for matching can bedesignated, according to the orientation of the face in a capturedimage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for describing an example of a matching supportapparatus.

FIG. 2 is a diagram for describing a user interface that is used inmatching support.

FIG. 3 is a diagram for describing an example of a system having thematching support apparatus.

FIG. 4 is a diagram for describing an example of candidate face imagedisplay.

FIG. 5 is a diagram for describing an example of operations of thematching support apparatus.

FIG. 6 is a diagram for describing an example of a system having thematching support apparatus.

FIG. 7 is a diagram for describing a user interface that is used inmatching support.

FIG. 8 is a diagram for describing an example of display in which afeature region on a reference face three-dimensional image is convertedto a feature region on a face image in a reference face developmentimage.

FIG. 9 is a diagram for describing an example of candidate face imagedisplay.

FIG. 10 is a diagram for describing an example of candidate face imagedisplay.

FIG. 11 is a diagram for describing an example of operations of thematching support apparatus.

FIG. 12 is a diagram for describing an example of a computer thatrealizes the matching support apparatus.

Example Embodiments

(Example Embodiment)

Hereinafter, an example embodiment of the invention will be describedwith reference to FIGS. 1 to 12.

[Apparatus Configuration]

Initially, the configuration of a matching support apparatus 1 in thisexample embodiment will be described using FIGS. 1 and 2. FIG. 1 is adiagram for describing an example of the matching support apparatus.FIG. 2 is a diagram for describing a user interface that is used inmatching support.

The matching support apparatus 10 shown in FIG. 1 is an apparatus withwhich a feature visible on the skin surface of the face of a persontargeted for matching can be designated, according to the orientation ofthe face in a captured image. Also, as shown in FIG. 1, the matchingsupport apparatus 10 has a first display information generation unit 11,a second display information generation unit 12 and a user interfacedisplay information generation unit 13. Furthermore, as shown in FIG. 1,the matching support apparatus 10 is connected to an image capturingapparatus 20, a storage device 21, an input device 22 and a displaydevice 30.

Of these, the first display information generation unit 11 generatesfirst display information for displaying, on the screen of the displaydevice 30, a first display region for displaying a matching image thatincludes the face of the person targeted for matching captured using theimage capturing apparatus 20.

The first display region is an area for displaying a matching image 41on a user interface 40 shown in FIG. 2, for example. The matching image41 is, for instance, a frame image of a still image or moving image.Note that, in the example in FIG. 2, the matching image 41 is displayedin a window 42. The configuration of the display screen is, however, notlimited to that in FIG. 2.

The second display information generation unit 12 generates seconddisplay information for displaying, on the screen of the display device30, a second display region for displaying a reference face developmentimage generated based on three-dimensional data of a head serving as areference.

The reference head is, for instance, a head created by CG (ComputerGraphics) based on data of one or more heads measured or captured in thepast. The reference head may also be created based on the head of aspecific person measured or captured in the past.

The second display region is an area for displaying a reference facedevelopment image 43 on the user interface 40 shown in FIG. 2, forexample. The reference face development image 43 is a development imageof a face cylindrically projected by executing UV developmentprocessing, for example, using the three-dimensional data of thereference head stored in the storage device 21 in advance. The creationof the development image of a face is, however, not limited to theabove-described cylindrical projection. Note that, in the example inFIG. 2, the reference face development image 43 is displayed in a window44. The configuration of the display screen is, however, not limited tothat in FIG. 2.

The user interface display information generation unit 13 generatesfirst user interface display information for displaying, on the screenof the display device 30, a first user interface to be used in anoperation for enabling the user to designate, in the second displayregion, a feature region indicating a facial feature of the persontargeted for matching visible on the skin surface of the person targetedfor matching.

Features on a person's face are sites indicating features of the personthat are visible on the skin surface such as moles, freckles, tattoos,birthmarks, wrinkles, dimples, scars, warts, lumps, rough skin anddiscolored skin patches, for example. In the example in FIG. 2, there isa mole 51 on the left cheek of the person captured in the matching image41, and thus this mole 51 is a feature.

The feature region is a region corresponding to a feature on theperson's face recognized by the user, with a marker that the userattaches to the reference face development image after having recognizedthe feature. In the example in FIG. 2, the region corresponding to themarker (x) on the reference face development image 43 is a featureregion 52. Note that a region of a person's face in which there are nofeatures may also be taken as a feature region.

In the example in FIG. 2, the first user interface corresponds to theuser interface 40. For example, in the case where the feature region 52is drawn (designated) using the user interface 40 and the input device22, which is a physical user interface such as a mouse, touch panel orkeyboard, the drawn feature region 52 is added to the reference facedevelopment image 43 when an “add” button 45 displayed on the userinterface 40 is selected.

In this way, in this example embodiment, by the user viewing thematching image 41 and designating the feature region 52 on the referenceface development image 43, matching support leading to specification ofthe person targeted for matching can be provided using the designatedfeature region 52, even if the person in the matching image 41 is notfacing forward. Matching support is processing for selecting a personhaving a feature in the same position as the person targeted formatching, using the feature region 52 designated on the reference facedevelopment image 43.

Also, by the feature region 52 being designated utilizing the referenceface development image 43, the influence of the undulations of the facearising from a change in the face orientation can be reduced, thusenabling the user to easily designate the feature region 52corresponding to the mole 51 on the reference face development image 43,even if the apparent position of the mole 51 changes.

[System Configuration]

Next, the configuration of the matching support apparatus 10 in thisexample embodiment will be described more specifically using FIG. 3.FIG. 3 is a diagram for describing an example of a system having thematching support apparatus.

As shown in FIG. 3, the system in this example embodiment has thematching support apparatus 10, the image capturing apparatus 20, thestorage device 21, the input device 22 and the display device 30. Forexample, the system is conceivably a monitoring system or anauthentication system. Also, the matching support apparatus 10 in FIG. 3has a candidate face image display information generation unit 14, ageneration unit 61, a matching unit 62, a selection unit 63, a detectionunit 64 and an association unit 65, in addition to the first displayinformation generation unit 11, the second display informationgeneration unit 12 and the user interface display information generationunit 13.

The system will now be described.

The matching support apparatus 10 is an information processing apparatussuch as a server computer, personal computer or mobile terminal equippedwith a CPU (Central Processing Unit), an FPGA (Field-Programmable GateArray) or both thereof, for example.

The image capturing apparatus 20 is an apparatus for capturing an imageof the face of the person targeted for matching. Specifically, the imagecapturing apparatus 20 transmits the captured image to the matchingsupport apparatus 10 via a communication network. The image capturingapparatus 20 is an image capturing apparatus such as a camera, forexample.

The storage device 21 stores the three-dimensional data of the referencehead described above and matching information. The matching informationis information in which a face development image for use in matching isassociated with a feature region for use in matching. The matching-useface development image is a face development image generated based onthe three-dimensional data of the head of each of a plurality of personsregistered in advance. The matching-use feature region is a featureregion indicating a feature visible on the skin surface of the head ofeach of the plurality of persons registered in advance. Note that thestorage device 21 may also store a reference face development image, areference face three-dimensional image or both thereof in advance.

Specifically, in the case of displaying the reference face developmentimage in the second display region, the storage device 21 transmits thethree-dimensional data of the reference head to the matching supportapparatus 10 via the communication network. Note that, in the case wherea reference face development image is stored, the storage device 21transmits the reference face development image to the matching supportapparatus 10.

Also, in the case where a person to serve as a candidate is selected,the storage device 21 transmits the matching information in which thematching-use face development image and matching-use feature regioncorresponding to the selected person are associated with each other tothe matching support apparatus 10 via the communication network.

Note that the storage device 21 is a storage device such as a database,for example. Also, information such as the three-dimensional data of thereference head and matching information described above may be storedseparately in a plurality of storage devices. Also, the storage device21 may be provided inside the matching support apparatus 10 or may beprovided externally thereto.

The input device 22 is a physical user interface such as a mouse, atouch panel or a keyboard, for example. Specifically, the input device22 is used by the user when providing matching support using a userinterface displayed on the display device 30.

The display device 30 acquires various display information and displaysgenerated images and the like on the screen, based on the acquireddisplay information. The display device 30 is a device that uses liquidcrystals, organic EL (Electroluminescence) or CRTs (Cathode Ray Tubes),for example. Furthermore, the display device 30 may also include anaudio output device such as a speaker. Note that the display device 30may also be a printing device such as a printer.

The matching support apparatus will now be described.

The first display information generation unit 11 generates first displayinformation for displaying, on the screen of the display device 30, afirst display region for displaying a matching image captured using theimage capturing apparatus 20. Specifically, the first displayinformation generation unit 11 acquires an image of a person captured bythe image capturing apparatus 20. Then, the first display informationgeneration unit 11 generates first display information for displaying,on the screen of the display device 30, a matching image 41 such asshown in FIG. 2, based on the acquired image. Thereafter, the firstdisplay information generation unit 11 transmits the first displayinformation to the display device 30.

Note that a configuration may be adopted in which only a frame image inwhich a feature is readily visible is used as the matching image. Also,a frame image in which a feature that was not visible until theorientation of the face changed is detected may be used as the matchingimage.

The second display information generation unit 12 generates seconddisplay information for displaying, on the screen of the display device30, a second display region for displaying a reference face developmentimage, based on the three-dimensional data of the reference head.Specifically, the second display information generation unit 12 acquiresthe three-dimensional data of the reference head from the storage device21.

Then, the second display information generation unit 12 generates areference face development image using the three-dimensional data of thereference head. Then, the second display information generation unit 12generates second display information for displaying, on the screen ofthe display device 30, a reference face development image 43 such asshown in FIG. 2, based on the generated reference face developmentimage. Thereafter, the second display information generation unit 12transmits the second display information to the display device 30. Notethat, in the case where a reference face development image is stored inthe storage device 21, the second display information generation unit 12may acquire the reference face development image directly from thestorage device 21.

The user interface display information generation unit 13 generatesfirst user interface display information for displaying, on the screenof the display device 30, a first user interface for enabling the userto designate a feature region in the second display region withreference to the first display region. Specifically, the user interfacedisplay information generation unit 13 displays a user interface 40 suchas shown in FIG. 2 as the first user interface to enable the user todesignate a feature region on the reference face development image.

Note that the user may designate a plurality of feature regions on thereference face development image, while viewing one matching image.Also, the user may designate one or more feature regions on thereference face development image, while viewing a plurality of matchingimages (frame images) in which the face is oriented differently. Forexample, the user may designate one or more feature regions on thereference face development image, while viewing a plurality of matchingimages in which the face is oriented differently. This results infeature regions being positioned accurately and enables the number offeature regions to be increased, thus improving matching accuracy.

Also, as shown in FIG. 2, it is conceivable to provide an “add” button45, a “delete” button 46, a “save” button 47, an “extract feature”button 48 and an “enlarge/reduce” button 49, for example. When the “add”button 45 is selected, the feature region 52 drawn on the reference facedevelopment image 43 can be added. When the “delete” button 46 isselected, the feature region 52 drawn on the reference face developmentimage 43 can be deleted. When the “save” button 47 is selected, featureinformation (e.g., texture information, position information, sizeinformation, shape information, feature type information indicating thetype of feature, etc.) relating to the designated feature region 52 isstored in a storage unit. When the “extract feature” button 48 isselected, a feature is automatically extracted from the matching image41. When the “enlarge/reduce” button 49 is selected, display of thematching image 41 or reference face development image 43 that isselected is enlarged or reduced. The editing functions are, however, notlimited to the above-described functions.

In the case where a feature region indicating a facial feature of theperson targeted for matching visible on the skin surface of the persontargeted for matching is designated, using the second display region,displayed on the screen of the display device 30, that is for displayingthe reference face development image generated based on thethree-dimensional data of the reference head, the generation unit 61generates feature information relating to that feature region.

Specifically, in the case where one or more feature regions aredesignated on the reference face development image, using the first userinterface, the generation unit 61, first, generates feature informationfor each designated feature region. Thereafter, the generation unit 61outputs the feature information to the matching unit 62.

In the example in FIG. 2, in the case where the feature region 52 isdesignated on the reference face development image 43 using the userinterface 40, the generation unit 61 generates feature information ofthe designated feature region 52.

The matching unit 62 matches the feature information against matchinginformation in which a matching-use face development image andmatching-use feature region for each person registered in advance areassociated with each other. Specifically, the matching unit 62, first,acquires feature information from the generation unit 61. Then, thematching unit 62 executes matching processing, with reference to therespective matching information stored in the storage device 21 usingthe acquired feature information, and calculates a matching result.Thereafter, the matching unit 62 associates the matching informationwith the calculated matching result.

The matching processing involves calculating a matching index (score) asa matching result, using a value indicating the approximateness of theposition of the designated feature region to the position of thematching-use feature region, or a deviation (distance) between theposition of the designated feature region and the position of thematching-use feature region, or a combination thereof, for example.Furthermore, in the case where a plurality of feature regions aredesignated, the matching index may be calculated using theinterpositional relationship between the plurality of feature regionsand the interpositional relationship between the plurality ofmatching-use feature regions.

The Euclidean distance between the two position coordinates of thedesignated feature region and the matching-use feature region, thesimilarity obtained through normalized correlation of textureinformation of the two feature regions, or the overlapping area of thetwo feature regions, for example, can be used as the matching index.

The selection unit 63 selects a person to serve as a candidate based ona matching result. Specifically, the selection unit 63 selects matchinginformation whose matching index is greater than or equal to a thresholdvalue set in advance. Then, the selection unit 63 outputs the selectedmatching information to the candidate face image display informationgeneration unit 14.

In the case where the feature region designated using the first userinterface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and acandidate person is selected based on a matching result, the candidateface image display information generation unit 14 generates candidateface image display information for displaying, on the screen of thedisplay device 30, the matching-use face development image correspondingto the selected candidate person together with the matching-use featureregion.

A specific description will now be given using FIG. 4. FIG. 4 is adiagram for describing an example of candidate face image display.

In the case where a candidate person is selected by the selection unit63 based on a matching result of the matching unit 62, the candidateface image display information generation unit 14, first, acquires, fromthe storage device 21, the matching information in which thematching-use face development image and matching-use feature regioncorresponding to the selected candidate person are associated with eachother.

Then, the candidate face image display information generation unit 14generates candidate face image display information for displaying, onthe screen of the display device 30, a matching-use face developmentimage 71 and matching-use feature region 73 such as shown in a window 72in FIG. 4, based on the acquired matching information. Thereafter, thecandidate face image display information generation unit 14 transmitsthe candidate face image display information to the display device 30.

Note that, in the case where there are a plurality of candidate persons,the matching-use face development images are displayed in order,according to the matching results. For example, the matching-use facedevelopment images are displayed in descending order of matching indices(scores) indicated by the matching results.

The detection unit 64 automatically detects a feature region from animage, displayed on the screen of the display device 30, that includesthe face of the person targeted for matching. Specifically, thedetection unit 64 automatically detects a feature region correspondingto a feature on the person's face (e.g., feature of the person visibleon the skin surface such as a mole, freckles, tattoo, birthmark,wrinkles, dimple, scar, wart, lump, skin roughness, discolored skinpatch, etc.), using a matching image. Use of a technique such assegmentation processing is conceivable for detecting a feature region.

The association unit 65 may automatically associate the position of thedetected feature region with a corresponding position on the referenceface development image. Association may involve automaticallyassociating the position of the detected feature region with acorresponding position on the reference face development image on thebasis of the relative positional relationship between parts of the facesuch as the eyes, nose and mouth, for example.

[Apparatus Operations]

Next, operations of the matching support apparatus in the exampleembodiment of the invention will be described using FIG. 5. FIG. 5 is adiagram for describing an example of operations of the matching supportapparatus. In the following description, FIGS. 1 to 4 will be referredto as appropriate. Also, in this example embodiment, a matching supportmethod is implemented by operating the matching support apparatus.Therefore, a description of the matching support method in this exampleembodiment is replaced by the following description of operations of thematching support apparatus.

As shown in FIG. 5, initially, the first display information generationunit 11 generates first display information for displaying, on thescreen of the display device 30, a first display region for displaying amatching image captured using the image capturing apparatus 20 (stepA1). Specifically, in step A1, the first display information generationunit 11 acquires an image of a person captured by the image capturingapparatus 20.

Then, in step A1, the first display information generation unit 11generates first display information for displaying, on the screen of thedisplay device 30, a matching image 41 such as shown in FIG. 2, based onthe acquired image. Thereafter, in step A1, the first displayinformation generation unit 11 transmits the first display informationto the display device 30.

Note that a configuration may be adopted in which only a frame image inwhich a feature is readily visible is used as the matching image. Also,a frame image in which a feature that was not visible until theorientation of the face changed is detected may be used as the matchingimage.

Also, the second display information generation unit 12 generates seconddisplay information for displaying, on the screen of the display device30, a second display region for displaying a reference face developmentimage, based on the three-dimensional data of the reference head (stepA2). Specifically, in step A2, the second display information generationunit 12 acquires the three-dimensional data of the reference head fromthe storage device 21.

Then, in step A2, the second display information generation unit 12generates a reference face development image using the three-dimensionaldata of the reference head. Then, in step A2, the second displayinformation generation unit 12 generates second display information fordisplaying, on the screen of the display device 30, a reference facedevelopment image 43 such as shown in FIG. 2, based on the generatedreference face development image. Thereafter, in step A2, the seconddisplay information generation unit 12 transmits the second displayinformation to the display device 30.

Note that, in the case where a reference face development image isstored in the storage device 21, the second display informationgeneration unit 12 may acquire the reference face development imagedirectly from the storage device 21.

The order of the above-described processing of A1 and processing of stepA2 may be reversed or the respective processing may be executed inparallel.

Next, the user interface display information generation unit 13generates first user interface display information for displaying, onthe screen of the display device 30, a first user interface for enablingthe user to designated a feature region in the second display regionwith reference to the first display region (step A3).

Specifically, in step A3, the user interface display informationgeneration unit 13 displays a user interface 40 such as shown in FIG. 2as the first user interface to enable the user to designate a featureregion on the reference face development image.

Note that the user may designate a plurality of feature regions on thereference face development image, while viewing one matching image.Also, the user may designate one or more feature regions on thereference face development image, while viewing a plurality of matchingimages (frame images) in which the face is oriented differently. Forexample, the user may designate one or more feature regions on thereference face development image, while viewing a plurality of matchingimages in which the face is oriented differently. This results infeature regions being positioned accurately, and enables the number offeature regions to be increased.

Furthermore, designation of a feature region may be performedautomatically. In that case, the detection unit 64 automatically detectsa feature region from an image, displayed on the screen of the displaydevice 30, that includes the face of the person targeted for matching,and the association unit 65 automatically associates the position of thedetected feature region with a corresponding position on the referenceface development image.

Next, in the case where a feature region indicating a facial feature ofthe person targeted for matching visible on the skin surface of theperson targeted for matching is designated, using the second displayregion, displayed on the screen of the display device 30, that is fordisplaying the reference face development image generated based on thethree-dimensional data of the reference head, the generation unit 61generates feature information relating to that feature region (step A4).

Specifically, in the case where one or more feature regions aredesignated on the reference face development image using the first userinterface, the generation unit 61, in step A4, first, generates featureinformation for each designated feature region. Thereafter, in step A4,the generation unit 61 outputs the feature information to the matchingunit 62.

In the example in FIG. 2, in the case where the feature region 52 isdesignated on the reference face development image 43 using the userinterface 40, the generation unit 61 generates feature information ofthe designated feature region 52.

The matching unit 62 matches the feature information against matchinginformation in which a matching-use face development image andmatching-use feature region for each person registered in advance areassociated with each other (step A5). Specifically, in step A5, thematching unit 62, first, acquires feature information from thegeneration unit 61. Then, in step A5, the matching unit 62 executesmatching processing with reference to respective matching informationstored in the storage device 21 using the acquired feature information,and calculates a matching result. Thereafter, in step A5, the matchingunit 62 associates the matching information with the calculated matchingresult.

The selection unit 63 selects a person to serve as a candidate based ona matching result (step A6). Specifically, in step A6, the selectionunit 63 selects matching information whose matching index is greaterthan or equal to a threshold value set in advance. Then, in step A6, theselection unit 63 outputs the selected matching information to thecandidate face image display information generation unit 14.

In the case where the feature region designated using the first userinterface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and acandidate person is selected based on a matching result, the candidateface image display information generation unit 14 generates candidateface image display information for displaying, on the screen of thedisplay device 30, the matching-use face development image correspondingto the selected candidate person together with the matching-use featureregion (step A7).

Specifically, in the case where a candidate person is selected by theselection unit 63 based on a matching result of the matching unit 62,the candidate face image display information generation unit 14, in stepA7, first, acquires, from the storage device 21, the matchinginformation in which the matching-use face development image andmatching-use feature region corresponding to the selected candidateperson are associated with each other.

Then, in step A7, the candidate face image display informationgeneration unit 14 generates candidate face image display informationfor displaying, on the screen of the display device 30, a matching-useface development image 71 and matching-use feature region 73 such asshown in window 72 in FIG. 4, based on the acquired matchinginformation. Thereafter, the candidate face image display informationgeneration unit 14 transmits the candidate face image displayinformation to the display device 30.

Note that, in the case where there are a plurality of candidate persons,the matching-use face development images are displayed in order,according to the matching results. For example, the matching-use facedevelopment images are displayed in descending order of matching indices(scores) indicated by the matching results.

Next, the display device 30 acquires the candidate face image displayinformation and displays a matching-use face development image 71 andmatching-use feature region 73 such as shown in window 72 in FIG. 4 onthe screen (step A8).

[Effects of Example Embodiment]

According to this example embodiment as described above, by the userviewing a matching image and designating a feature region on a referenceface development image, matching support leading to specification of theperson targeted for matching can be provided using the designatedfeature region, even if the person in the matching image is not facingforward.

Also, by the feature region being designated utilizing the referenceface development image, the influence of the undulations of the facearising from a change in the face orientation can be reduced, thusenabling the feature region corresponding to the feature to be easilydesignated on the reference face development image by the user, even ifthe apparent position of the feature changes.

Also, a matching apparatus that extracts an image corresponding to aperson from a frame image captured by an image capturing apparatus andexecutes matching using the extracted image may be coordinated with thematching support apparatus. In that case, if an image corresponding tothe feature is detected from any of the frame images, the processing maybe switched from a matching processing mode for performing matchingusing the matching apparatus to a matching support processing mode forproviding matching support using the matching support apparatus.

Also, if it is judged that the person in the captured image is the sameas a person registered in advance, the matching-use face developmentimage and matching-use feature region corresponding to the person in thecaptured image may be edited, based on the feature of the person.

Also, if it is judged that the person in the captured image is not thesame as a person registered in advance, the matching-use facedevelopment image and matching-use feature region corresponding to theperson in the captured image may be edited, based on the feature showingthe difference between the person in the captured image and the personregistered in advance.

[Program]

A program in the example embodiment of the invention need only be aprogram for causing a computer to execute the processing from step A1shown in FIG. 5. The matching support apparatus and matching supportmethod of this example embodiment can be realized, by this program beinginstalled on a computer and executed. In this case, a processor of thecomputer functions and performs processing as the first displayinformation generation unit 11, the second display informationgeneration unit 12, the user interface display information generationunit 13, the candidate face image display information generation unit14, the generation unit 61, the matching unit 62, the selection unit 63,the detection unit 64 and the association unit 65.

Also, the program in this example embodiment may be executed by acomputer system constructed from a plurality of computers. In this case,for example, the computers may each function as one of the first displayinformation generation unit 11, the second display informationgeneration unit 12, the user interface display information generationunit 13, the candidate face image display information generation unit14, the generation unit 61, the matching unit 62, the selection unit 63,the detection unit 64 and the association unit 65.

(Example Variation)

Hereinafter, an example variation of the invention will be describedwith reference to FIGS. 6 to 11.

[System Configuration]

The example variation will be described using FIG. 6. FIG. 6 is adiagram for describing an example of a system having the matchingsupport apparatus.

As shown in FIG. 6, the matching support apparatus 10 in the examplevariation has a third display information generation unit 15 and aconversion unit 66, in addition to the first display informationgeneration unit 11, the second display information generation unit 12,the user interface display information generation unit 13, the candidateface image display information generation unit 14, the generation unit61, the matching unit 62, the selection unit 63, the detection unit 64and the association unit 65.

In the example variation, in the case of further displaying a referenceface three-dimensional image in a third display region, the storagedevice 21 transmits the three-dimensional data of the reference head tothe matching support apparatus 10, via the communication network. Thethird display region will be described later. Note that, in the casewhere a reference face three-dimensional image is stored, the storagedevice 21 transmits the reference face development image to the matchingsupport apparatus 10.

The third display information generation unit 15 generates third displayinformation for displaying, on the screen of the display device 30, athird display region in which the face in a reference facethree-dimensional image generated based on the three-dimensional data ofthe reference head is displayed to be oriented in alignment with theorientation of the face of the person targeted for matching that isdisplayed in the first display region. Specifically, the third displayinformation generation unit 15 acquires the three-dimensional data ofthe reference head from the storage device 21.

Then, the third display information generation unit 15 generates areference face three-dimensional image using the three-dimensional dataof the reference head. Then, the third display information generationunit 15 generates third display information for displaying, on thescreen of the display device 30, a reference face three-dimensionalimage 81 such as shown in FIG. 6, based on the generated reference facethree-dimensional image. Thereafter, the third display informationgeneration unit 15 transmits the third display information to thedisplay device 30. Note that, in the case where a reference facethree-dimensional image is stored in the storage device 21, the thirddisplay information generation unit 15 may acquire the reference facethree-dimensional image directly from the storage device 21.

Note that the orientation of the face may be manually aligned by theuser using the user interface, or may be automatically aligned.

In the example variation, the user interface display informationgeneration unit 13 generates second user interface display informationfor displaying, on the screen of the display device 30, a second userinterface to be used in an operation for enabling the user to designatea feature region in the third display region.

A specific description will now be given using FIG. 7. FIG. 7 is adiagram for describing a user interface that is used in matchingsupport.

The user interface display information generation unit 13 displays auser interface 80 such as shown in FIG. 7 as the second user interfaceto enable the user to designate a feature region on the reference facethree-dimensional image. Note that, in the example in FIG. 7, thereference face three-dimensional image 81 is displayed in a window 82.The configuration of the display screen is, however, not limited to thatin FIG. 7.

Note that the user may designate a plurality of feature regions on thereference face three-dimensional image, while viewing one matchingimage. Also, the user may designate one or more feature regions on thereference face three-dimensional image, while viewing a plurality ofmatching images (frame images) in which the face is orienteddifferently. For example, the user may designate one or more featureregions on the reference face three-dimensional image, while viewing aplurality of matching images in which the face is oriented differently.This results in feature regions being positioned accurately, and enablesthe number of feature regions to be increased.

Also, in the example variation, for example, when the “add” button 45 isselected after the user has selected the reference facethree-dimensional image 81 shown in FIG. 7, a feature region 83 drawn onthe reference face three-dimensional image 81 can be added. When the“delete” button 46 is selected, the feature region 83 drawn on thereference face three-dimensional image 81 can be deleted. When the“save” button 47 is selected, feature information (e.g., textureinformation, position information, feature type information, etc.)relating to the designated feature region 83 is stored in a storageunit. When the “extract feature” button 48 is selected, a feature isautomatically extracted from the matching image 41. When the“enlarge/reduce” button 49 is selected, display of the matching image 41or reference face development image 43 or reference facethree-dimensional image 81 that is selected is enlarged or reduced. Theediting functions are, however, not limited to the above-describedfunctions.

In the example variation, in the case where a feature region indicatinga facial feature of the person targeted for matching visible on the skinsurface of the person targeted for matching is designated, using a thirddisplay region, displayed on the screen of the display device 30, thatis for displaying a reference face three-dimensional image generatedbased on the three-dimensional data of the reference head, thegeneration unit 61 generates feature information relating to thatfeature region.

Specifically, in the case where one or more feature regions aredesignated on the reference face three-dimensional image using thesecond user interface, the generation unit 61, first, generates featureinformation for each designated feature region. Thereafter, thegeneration unit 61 outputs the feature information to the matching unit62. In the example in FIG. 7, in the case where the feature region 83 isdesignated on the reference face three-dimensional image 81 using theuser interface 80, the generation unit 61 generates feature informationof the designated feature region 83.

In the case where a feature region is designated using the referenceface three-dimensional image display region in which the face in thereference face three-dimensional image generated based on thethree-dimensional data of the reference head is displayed to be orientedin alignment with the orientation of the face of the person targeted formatching, the conversion unit 66 converts the feature informationrelating to the feature region designated on the reference facethree-dimensional image to feature information to be used with thereference face development image.

A description of the matching unit 62 and the selection unit 63 is givenabove and will thus be omitted here.

A specific description will now be given using FIG. 8. FIG. 8 is adiagram for describing an example of display in which a feature regionon a reference face three-dimensional image is converted to a featureregion on a face image in a reference face development image.

In FIG. 8, when the feature region 83 is added, a feature region 86corresponding to the feature region 83 is also added to a reference facedevelopment image 84. Note that, in the example in FIG. 8, the referenceface development image 84 is displayed in a window 85. The configurationof the display screen is, however, not limited to that in FIG. 8.

In the example variation, in the case where the feature regiondesignated using the second user interface is matched against matchinginformation in which a matching-use face development image andmatching-use feature region for each person registered in advance areassociated with each other, and a candidate person is selected based ona matching result, the candidate face image display informationgeneration unit 14 generates candidate face image display informationfor displaying, on the screen of the display device 30, the matching-useface development image corresponding to the selected candidate togetherwith the feature region.

A specific description will now be given using FIG. 9. FIG. 9 is adiagram for describing an example of candidate face image display.

In the case where a candidate person is selected by the selection unit63 based on a matching result of the matching unit 62, the candidateface image display information generation unit 14, first, acquires, fromthe storage device 21, the matching information in which thematching-use face development image and matching-use feature regioncorresponding to the selected candidate person are associated with eachother.

Then, the candidate face image display information generation unit 14generates candidate face image display information for displaying, onthe screen of the display device 30, a matching-use face developmentimage 91 and matching-use feature region 93 such as shown in a window 92in FIG. 9, based on the acquired matching information. Thereafter, thecandidate face image display information generation unit 14 transmitsthe candidate face image display information to the display device 30.

Note that, in the case where there are a plurality of candidate persons,the matching-use face development images are displayed in order,according to the matching results. For example, the matching-use facedevelopment images are displayed in descending order of matching indices(scores) indicated by the matching results.

Also, in the example variation, the candidate face image displayinformation generation unit 14 generates candidate facethree-dimensional image display information for displaying, on thescreen of the display device 30, the matching-use face three-dimensionalimage corresponding to the candidate person, based on thethree-dimensional data of the head of the selected candidate person.

A specific description will now be given using FIG. 10. FIG. 10 is adiagram for describing an example of candidate face image display.

In the case where a candidate person is selected by the selection unit63 based on a matching result of the matching unit 62, the candidateface image display information generation unit 14, first, acquires, fromthe storage device 21, the matching information in which thematching-use face three-dimensional image and matching-use featureregion corresponding to the selected candidate person are associatedwith each other.

Then, the candidate face image display information generation unit 14generates candidate face image display information for displaying, onthe screen of the display device 30, a matching-use facethree-dimensional image 91′ and matching-use feature region 93′ such asshown in a window 92′ in FIG. 10, based on the acquired matchinginformation. Thereafter, the candidate face image display informationgeneration unit 14 transmits the candidate face image displayinformation to the display device 30.

Note that one or more of the above-described matching image, referenceface development image, reference face three-dimensional image,matching-use face development image including a matching-use featureregion and matching-use face three-dimensional image including amatching-use feature region may be combined for display on the userinterface.

[Apparatus Operations]

Operations of the example variation will be described using FIG. 11.FIG. 11 is a diagram for describing an example of operations of thematching support apparatus. In the following description, FIGS. 1 to 10will be referred to as appropriate. Also, in the example variation, amatching support method is implemented by operating the matching supportapparatus. Therefore, the description of the matching support method inthe example variation is replaced by the following description of theoperations of the matching support apparatus.

As shown in FIG. 11, initially, the processing of step A1 describedabove is executed.

The third display information generation unit 15 generates third displayinformation for displaying, on the screen of the display device 30, athird display region in which the face in a reference facethree-dimensional image generated based on the three-dimensional data ofthe reference head is displayed to be oriented in alignment with theorientation of the face of the person targeted for matching that isdisplayed in the first display region (step B1). Specifically, the thirddisplay information generation unit 15 acquires the three-dimensionaldata of the reference head from the storage device 21.

Then, in step B1, the third display information generation unit 15generates a reference face three-dimensional image using thethree-dimensional data of the reference head. Then, in step B1, thethird display information generation unit 15 generates third displayinformation for displaying, on the screen of the display device 30, areference face three-dimensional image 81 such as shown in FIG. 7, basedon the generated reference face three-dimensional image. Thereafter, instep B1, the third display information generation unit 15 transmits thethird display information to the display device 30.

Note that, in the case where a reference face three-dimensional image isstored in the storage device 21, the third display informationgeneration unit 15 may acquire the reference face three-dimensionalimage directly from the storage device 21.

Note that the orientation of the face may be manually aligned by theuser, or may be automatically aligned.

The order of the processing of step A1 and the processing of step B1described above may be reversed, or the respective processing may beexecuted in parallel.

Next, the user interface display information generation unit 13generates second user interface display information for displaying, onthe screen of the display device 30, a second user interface to be usedin an operation for enabling the user to designate a feature region inthe third display region (step B2).

Specifically, in step B2, the user interface display informationgeneration unit 13 displays a user interface 80 such as shown in FIG. 7as the second user interface to enable the user to designate a featureregion on the reference face three-dimensional image. Note that, in theexample in FIG. 7, the reference face three-dimensional image 81 isdisplayed in the window 82. The configuration of the display screen is,however, not limited to that in FIG. 7.

Note that the user may designate a plurality of feature regions on thereference face three-dimensional image, while viewing one matchingimage. Also, the user may designate one or more feature regions on thereference face three-dimensional image, while viewing a plurality ofmatching images (frame images) in which the face is orienteddifferently. For example, the user may designate one or more featureregions on the reference face three-dimensional image, while viewing aplurality of matching images in which the face is oriented differently.This results in feature regions being positioned accurately and enablesthe number of feature regions to be increased, thus improving matchingaccuracy.

Furthermore, designation of a feature region may be performedautomatically. In that case, the detection unit 64 automatically detectsa feature region from an image, displayed on the screen of the displaydevice 30, that includes the face of the person targeted for matching,and the association unit 65 automatically associates the position of thedetected feature region with a corresponding position on the referenceface three-dimensional image.

Next, in the case where a feature region indicating a facial feature ofthe person targeted for matching visible on the skin surface of theperson targeted for matching is designated, using a third displayregion, displayed on the screen of the display device 30, that is fordisplaying the reference face three-dimensional image generated based onthe three-dimensional data of the reference head, the generation unit 61generates feature information relating to that feature region (step B3).

Specifically, in the case where one or more feature regions aredesignated on the reference face three-dimensional image, using thesecond user interface, the generation unit 61, in step B3, first,generates feature information for each designated feature region.Thereafter, in step B3, the generation unit 61 outputs the featureinformation to the matching unit 62. In the example in FIG. 7, in thecase where the feature region 83 is designated on the reference facethree-dimensional image 81 using the user interface 80, the generationunit 61 generates feature information of the designated feature region83.

Next, in the case where a feature region is designated using thereference face three-dimensional image display region in which the facein the reference face three-dimensional image generated based on thethree-dimensional data of the reference head is displayed to be orientedin alignment with the orientation of the face of the person targeted formatching, the conversion unit 66 converts the feature informationrelating to the feature region designated on the reference facethree-dimensional image to feature information to be used with thereference face development image (step B4).

Next, the processing of steps A5 and A6 described above is executed.

Next, in the case where the feature region designated using the seconduser interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and acandidate person is selected based on a matching result, the candidateface image display information generation unit 14 generates candidateface image display information for displaying, on the screen of thedisplay device 30, the matching-use face development image correspondingto the selected candidate together with the feature region (step B5).

In the case where a candidate person is selected by the selection unit63 based on a matching result of the matching unit 62, the candidateface image display information generation unit 14, in step B5, first,acquires, from the storage device 21, the matching information in whichthe matching-use face development image and matching-use feature regioncorresponding to the selected candidate person are associated with eachother.

Then, in step B5, the candidate face image display informationgeneration unit 14 generates candidate face image display informationfor displaying, on the screen of the display device 30, a matching-useface development image 91 and matching-use feature region 93 such asshown in the window 92 in FIG. 9, based on the acquired matchinginformation. Thereafter, the candidate face image display informationgeneration unit 14 transmits the candidate face image displayinformation to the display device 30.

Note that, in the case where there are a plurality of candidate persons,the matching-use face development images are displayed in order,according to the matching results. For example, the matching-use facedevelopment images are displayed in descending order of matching indices(scores) indicated by the matching results.

Also, in step B5, the candidate face image display informationgeneration unit 14 may generate candidate face three-dimensional imagedisplay information for displaying, on the screen of the display device30, the matching-use face three-dimensional image corresponding to thecandidate person, based on the three-dimensional data of the head of theselected candidate person.

In the case where a candidate person is selected by the selection unit63 based on a matching result of the matching unit 62, the candidateface image display information generation unit 14, in step B5, first,acquires, from the storage device 21, the matching information in whichthe matching-use face three-dimensional image and matching-use featureregion corresponding to the selected candidate person are associatedwith each other.

Then, in step B5, the candidate face image display informationgeneration unit 14 generates candidate face image display informationfor displaying, on the screen of the display device 30, a matching-useface three-dimensional image 91′ and matching-use feature region 93′such as shown in the window 92′ in FIG. 10, based on the acquiredmatching information. Thereafter, the candidate face image displayinformation generation unit 14 transmits the candidate face imagedisplay information to the display device 30.

Note that, in the case where there are a plurality of candidate persons,the matching-use face development images are displayed in order,according to the matching results. For example, the matching-use facedevelopment images are displayed in descending order of matching indices(scores) indicated by the matching results.

Next, the display device 30 acquires the candidate face image displayinformation and displays a matching-use face development image 91 andmatching-use feature region 93 such as shown in the window 92 in FIG. 9on the screen (step B6). Alternatively, in step B6, the display device30 may display a matching-use face development image 91′ andmatching-use feature region 93′ such as shown in the window 92′ in FIG.10 on the screen.

[Effects of Variation]

According to the example variation as described above, by the userviewing a matching image and designating a feature region on a referenceface three-dimensional image, matching support leading to specificationof the person targeted for matching can be provided using the designatedfeature region, even if the person in the matching image is not facingforward.

Also, by the feature region being designated utilizing the referenceface three-dimensional image, the influence of the undulations of theface arising from a change in the face orientation can be reduced, thusenabling the feature region corresponding to the feature to be easilydesignated on the reference face three-dimensional image by the user,even if the apparent position of the feature changes.

Also, a matching apparatus that extracts an image corresponding to aperson from a frame image captured by an image capturing apparatus andexecutes matching using the extracted image may be coordinated with thematching support apparatus. In that case, if an image corresponding tothe feature is detected from any of the frame images, the processing maybe switched from a matching processing mode for performing matchingusing the matching apparatus that is currently set to a matching supportprocessing mode for providing matching support using the matchingsupport apparatus.

Also, if it is judged that the person in the captured image is the sameas a person registered in advance, the matching-use face developmentimage and matching-use feature region corresponding to the person in thecaptured image may be edited, based on the feature of the person.

Also, if it is judged that the person in the captured image is not thesame as a person registered in advance, the matching-use facedevelopment image and matching-use feature region corresponding to theperson in the captured image may be edited, based on the feature showingthe difference between the person in the captured image and the personregistered in advance.

[Program]

A program in the example variation of the invention need only be aprogram for causing a computer to execute the processing from step A1shown in FIG. 11. The matching support apparatus and matching supportmethod of this example embodiment can be realized, by this program beinginstalled on a computer and executed. In this case, a processor of thecomputer functions and performs processing as the first displayinformation generation unit 11, the second display informationgeneration unit 12, the user interface display information generationunit 13, the candidate face image display information generation unit14, the third display information generation unit 15, the generationunit 61, the matching unit 62, the selection unit 63, the detection unit64, the association unit 65 and the conversion unit 66.

Also, the program in this example embodiment may be executed by acomputer system constructed from a plurality of computers. In this case,for example, the computers may each function as one of the first displayinformation generation unit 11, the second display informationgeneration unit 12, the user interface display information generationunit 13, the candidate face image display information generation unit14, the third display information generation unit 15, the generationunit 61, the matching unit 62, the selection unit 63, the detection unit64, the association unit 65 and the conversion unit 66.

[Physical Configuration]

Here, a computer that realizes the matching support apparatus byexecuting programs of the example embodiment and variation will bedescribed using FIG. 12. FIG. 12 is a block diagram showing an exampleof a computer that realizes the matching support apparatus of theexample embodiment and variation of the invention.

As shown in FIG. 12, a computer 110 includes a CPU 111, a main memory112, a storage device 113, an input interface 114, a display controller115, a data reader/writer 116 and a communication interface 117. Theseconstituent elements are connected to each other in a data communicablemanner via a bus 121. Note that the computer 110 may also include a GPU(Graphics Processing Unit) or FPGA, in addition to the CPU 111 orinstead of the CPU 111.

The CPU 111 carries out various computational operations by extractingprograms (code) of the example embodiment that are stored in the storagedevice 113 to the main memory 112 and executing these programs inpredetermined order. The main memory 112 is typically a volatile storagedevice such as a DRAM (Dynamic Random Access Memory). Also, programs ofthe example embodiment are provided in a state of being stored in acomputer-readable recording medium 120. Note that programs of theexample embodiment may also be distributed over the Internet connectedvia the communication interface 117. Note that the recording medium 120is a nonvolatile storage medium.

Also, specific examples of the storage device 113 include asemiconductor storage device such as a flash memory, in addition to ahard disk drive. The input interface 114 mediates data transmissionbetween the CPU 111 and an input device 118 such as a keyboard and amouse. The display controller 115 is connected to a display device 119and controls display on the display device 119.

The data reader/writer 116 mediates data transmission between the CPU111 and the recording medium 120, and executes readout of programs fromthe recording medium 120 and writing of processing results of thecomputer 110 to the recording medium 120. The communication interface117 mediates data transmission between the CPU 111 and other computers.

Also, specific examples of the recording medium 120 include ageneral-purpose semiconductor storage device such as a CF (Compact Flash(registered trademark) card or SD (Secure Digital) card, a magneticrecording medium such as a flexible disk, and an optical recordingmedium such as a CD-ROM (Compact Disk Read Only Memory).

Note that the matching support apparatus in the example embodiment canalso be realized by using hardware corresponding to the respectiveconstituent elements, rather than by a computer on which programs areinstalled. Furthermore, the matching support apparatus may be partiallyrealized by programs and the remaining portion thereof may be realizedby hardware.

The following supplementary notes will be further disclosed in relationto the above example embodiment. The example embodiment described abovecan be partially or wholly realized by supplementary notes 1 to 18described below, but the invention is not limited to the followingdescription.

(Supplementary Note 1)

A matching support apparatus including:

a first display information generation unit configured to generate firstdisplay information for displaying, on a screen of a display device, afirst display region for displaying an image including a face of aperson targeted for matching captured using an image capturingapparatus;

a second display information generation unit configured to generatesecond display information for displaying, on the screen of the displaydevice, a second display region for displaying a reference facedevelopment image generated based on three-dimensional data of a headserving as a reference; and

a user interface display information generation unit configured togenerate first user interface display information for displaying, on thescreen of the display device, a first user interface to be used in anoperation for enabling a user to designate, in the second displayregion, a feature region indicating a facial feature of the persontargeted for matching visible on a skin surface of the person targetedfor matching.

(Supplementary Note 2)

The matching support apparatus according to supplementary note 1,including:

a candidate face image display information generation unit configuredto, in a case where the feature region designated using the first userinterface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generate candidate face image display information for displaying, on thescreen of the display device, the matching-use face development imagecorresponding to the selected candidate person together with thematching-use feature region.

(Supplementary Note 3)

The matching support apparatus according to supplementary note 2,including:

a third display information generation unit configured to generate thirddisplay information for displaying, on the screen of the display device,a third display region in which a face in a reference facethree-dimensional image generated based on the three-dimensional data ofthe reference head is displayed to be oriented in alignment with anorientation of the face of the person targeted for matching displayed inthe first display region.

(Supplementary Note 4)

The matching support apparatus according to supplementary note 3,

whereby the user interface display information generation unit generatessecond user interface display information for displaying, on the screenof the display device, a second user interface to be used in anoperation for enabling the user to designate the feature region in thethird display region.

(Supplementary Note 5)

The matching support apparatus according to supplementary note 4,

whereby the candidate face image display information generation unit, ina case where the feature region designated using the second userinterface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generates candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thefeature region.

(Supplementary Note 6)

The matching support apparatus according to any one of supplementarynotes 2 to 5, whereby the candidate face image display informationgeneration unit generates candidate face three-dimensional image displayinformation for displaying, on the screen of the display device, amatching-use face three-dimensional image corresponding to the selectedcandidate person, based on three-dimensional data of a head of theselected candidate person.

(Supplementary Note 7)

A matching support method including:

(a) a step of generating first display information for displaying, on ascreen of a display device, a first display region for displaying animage including a face of a person targeted for matching captured usingan image capturing apparatus;

(b) a step of generating second display information for displaying, onthe screen of the display device, a second display region for displayinga reference face development image generated based on three-dimensionaldata of a head serving as a reference; and

(c) a step of generating first user interface display information fordisplaying, on the screen of the display device, a first user interfaceto be used in an operation for enabling a user to designate, in thesecond display region, a feature region indicating a facial feature ofthe person targeted for matching visible on a skin surface of the persontargeted for matching.

(Supplementary Note 8)

The matching support method according to supplementary note 7,including:

(d) a step of, in a case where the feature region designated using thefirst user interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thematching-use feature region.

(Supplementary Note 9)

The matching support method according to supplementary note 8,including:

(e) a step of generating third display information for displaying, onthe screen of the display device, a third display region in which a facein a reference face three-dimensional image generated based on thethree-dimensional data of the reference head is displayed to be orientedin alignment with an orientation of the face of the person targeted formatching displayed in the first display region.

(Supplementary Note 10)

The matching support method according to supplementary note 9,including:

(f) a step of generating second user interface display information fordisplaying, on the screen of the display device, a second user interfaceto be used in an operation for enabling the user to designate thefeature region in the third display region.

(Supplementary Note 11)

The matching support method according to supplementary note 10,including:

(g) a step of, in a case where the feature region designated using thesecond user interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thefeature region.

(Supplementary Note 12)

The matching support method according to any one of supplementary notes8 to 11, including:

(h) a step of generating candidate face three-dimensional image displayinformation for displaying, on the screen of the display device, amatching-use face three-dimensional image corresponding to the selectedcandidate person, based on three-dimensional data of a head of theselected candidate person.

(Supplementary Note 13)

A computer-readable recording medium that includes a program recordedthereon, the program including instructions that cause a computer tocarry out:

(a) a step of generating first display information for displaying, on ascreen of a display device, a first display region for displaying animage including a face of a person targeted for matching captured usingan image capturing apparatus;

(b) a step of generating second display information for displaying, onthe screen of the display device, a second display region for displayinga reference face development image generated based on three-dimensionaldata of a head serving as a reference; and

(c) a step of generating first user interface display information fordisplaying, on the screen of the display device, a first user interfaceto be used in an operation for enabling a user to designate, in thesecond display region, a feature region indicating a facial feature ofthe person targeted for matching visible on a skin surface of the persontargeted for matching.

(Supplementary Note 14)

The computer-readable recording medium according to supplementary note13, the program further including instructions that cause the computerto carry out:

(d) a step of, in a case where the feature region designated using thefirst user interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thematching-use feature region.

(Supplementary Note 15)

The computer-readable recording medium according to supplementary note14, the program further including instructions that cause the computerto carry out:

(e) a step of generating third display information for displaying, onthe screen of the display device, a third display region in which a facein a reference face three-dimensional image generated based on thethree-dimensional data of the reference head is displayed to be orientedin alignment with an orientation of the face of the person targeted formatching displayed in the first display region.

(Supplementary Note 16)

The computer-readable recording medium according to supplementary note15, the program further including instructions that cause the computerto carry out:

(f) a step of generating second user interface display information fordisplaying, on the screen of the display device, a second user interfaceto be used in an operation for enabling the user to designate thefeature region in the third display region.

(Supplementary Note 17)

The computer-readable recording medium according to supplementary note16, the program further including instructions that cause the computerto carry out:

(g) a step of, in a case where the feature region designated using thesecond user interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thefeature region.

(Supplementary Note 18)

The computer-readable recording medium according to any one ofsupplementary notes 14 to 17, the program further including instructionsthat cause the computer to carry out:

(h) a step of generating candidate face three-dimensional image displayinformation for displaying, on the screen of the display device, amatching-use face three-dimensional image corresponding to the selectedcandidate person, based on three-dimensional data of a head of theselected candidate person.

Although the instant invention has been described above with referenceto an example embodiment, the invention is not limited to the foregoingexample embodiment. Various modifications that will be appreciated bythose skilled in the art can be made to the configurations and detailsof the instant invention within the scope of the invention.

INDUSTRIAL APPLICABILITY

According to the invention as described above, a feature visible on theskin surface of the face of a person targeted for matching can bedesignated, according to the orientation of the face in a capturedimage. The invention is useful in fields that require matching such asmonitoring systems and authentication systems.

LIST OF REFERENCE SIGNS

10 Matching support apparatus

11 First display information generation unit

12 Second display information generation unit

13 User interface display information generation unit

14 Candidate face image display information generation unit

15 Third display information generation unit

20 Image capturing apparatus

21 Storage device

22 Input device

30 Display device

61 Generation unit

62 Matching unit

63 Selection unit

64 Detection unit

65 Association unit

66 Conversion unit

110 Computer

111 CPU

112 Main memory

113 Storage device

114 Input interface

115 Display controller

116 Data reader/writer

117 Communication interface

118 Input device

119 Display device

120 Recording medium

121 Bus

What is claimed is:
 1. A matching support apparatus comprising: at leastone memory configured to store instructions; and at least one processorconfigured to execute the instructions to: generate first displayinformation for displaying, on a screen of a display device, a firstdisplay region for displaying an image including a face of a persontargeted for matching captured using an image capturing apparatus;generate second display information for displaying, on the screen of thedisplay device, a second display region for displaying a reference facedevelopment image generated based on three-dimensional data of a headserving as a reference; and generate first user interface displayinformation for displaying, on the screen of the display device, a firstuser interface to be used in an operation for enabling a user todesignate, in the second display region, a feature region indicating afacial feature of the person targeted for matching visible on a skinsurface of the person targeted for matching.
 2. The matching supportapparatus according to claim 1, comprising: in a case where the featureregion designated using the first user interface is matched againstmatching information in which a matching-use face development image andmatching-use feature region for each person registered in advance areassociated with each other, and a person to serve as a candidate isselected based on a matching result, generating candidate face imagedisplay information for displaying, on the screen of the display device,the matching-use face development image corresponding to the selectedcandidate person together with the matching-use feature region.
 3. Thematching support apparatus according to claim 2, comprising: generatethird display information for displaying, on the screen of the displaydevice, a third display region in which a face in a reference facethree-dimensional image generated based on the three-dimensional data ofthe reference head is displayed to be oriented in alignment with anorientation of the face of the person targeted for matching displayed inthe first display region.
 4. The matching support apparatus according toclaim 3, generate second user interface display information fordisplaying, on the screen of the display device, a second user interfaceto be used in an operation for enabling the user to designate thefeature region in the third display region.
 5. The matching supportapparatus according to claim 4, in a case where the feature regiondesignated using the second user interface is matched against matchinginformation in which a matching-use face development image andmatching-use feature region for each person registered in advance areassociated with each other, and a person to serve as a candidate isselected based on a matching result, generate candidate face imagedisplay information for displaying, on the screen of the display device,the matching-use face development image corresponding to the selectedcandidate person together with the feature region.
 6. The matchingsupport apparatus according to claim 2, generate candidate facethree-dimensional image display information for displaying, on thescreen of the display device, a matching-use face three-dimensionalimage corresponding to the selected candidate person, based onthree-dimensional data of a head of the selected candidate person.
 7. Amatching support method comprising: generating first display informationfor displaying, on a screen of a display device, a first display regionfor displaying an image including a face of a person targeted formatching captured using an image capturing apparatus; generating seconddisplay information for displaying, on the screen of the display device,a second display region for displaying a reference face developmentimage generated based on three-dimensional data of a head serving as areference; and generating first user interface display information fordisplaying, on the screen of the display device, a first user interfaceto be used in an operation for enabling a user to designate, in thesecond display region, a feature region indicating a facial feature ofthe person targeted for matching visible on a skin surface of the persontargeted for matching.
 8. The matching support method according to claim7, comprising: in a case where the feature region designated using thefirst user interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thematching-use feature region.
 9. The matching support method according toclaim 8, comprising: generating third display information fordisplaying, on the screen of the display device, a third display regionin which a face in a reference face three-dimensional image generatedbased on the three-dimensional data of the reference head is displayedto be oriented in alignment with an orientation of the face of theperson targeted for matching displayed in the first display region. 10.The matching support method according to claim 9, comprising: generatingsecond user interface display information for displaying, on the screenof the display device, a second user interface to be used in anoperation for enabling the user to designate the feature region in thethird display region.
 11. The matching support method according to claim10, comprising: in a case where the feature region designated using thesecond user interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thefeature region.
 12. The matching support method according to claim 8,comprising: generating candidate face three-dimensional image displayinformation for displaying, on the screen of the display device, amatching-use face three-dimensional image corresponding to the selectedcandidate person, based on three-dimensional data of a head of theselected candidate person.
 13. A non-transitory computer-readablerecording medium that includes a program recorded thereon, the programincluding instructions that cause a computer to carry out: generatingfirst display information for displaying, on a screen of a displaydevice, a first display region for displaying an image including a faceof a person targeted for matching captured using an image capturingapparatus; generating second display information for displaying, on thescreen of the display device, a second display region for displaying areference face development image generated based on three-dimensionaldata of a head serving as a reference; and generating first userinterface display information for displaying, on the screen of thedisplay device, a first user interface to be used in an operation forenabling a user to designate, in the second display region, a featureregion indicating a facial feature of the person targeted for matchingvisible on a skin surface of the person targeted for matching.
 14. Thenon-transitory computer-readable recording medium according to claim 13,the program further including instructions that cause the computer tocarry out: in a case where the feature region designated using the firstuser interface is matched against matching information in which amatching-use face development image and matching-use feature region foreach person registered in advance are associated with each other, and aperson to serve as a candidate is selected based on a matching result,generating candidate face image display information for displaying, onthe screen of the display device, the matching-use face developmentimage corresponding to the selected candidate person together with thematching-use feature region.
 15. The non-transitory computer-readablerecording medium according to claim 14, the program further includinginstructions that cause the computer to carry out: generating thirddisplay information for displaying, on the screen of the display device,a third display region in which a face in a reference facethree-dimensional image generated based on the three-dimensional data ofthe reference head is displayed to be oriented in alignment with anorientation of the face of the person targeted for matching displayed inthe first display region.
 16. The non-transitory computer-readablerecording medium according to claim 15, the program further includinginstructions that cause the computer to carry out: generating seconduser interface display information for displaying, on the screen of thedisplay device, a second user interface to be used in an operation forenabling the user to designate the feature region in the third displayregion.
 17. The non-transitory computer-readable recording mediumaccording to claim 16, the program further including instructions thatcause the computer to carry out: in a case where the feature regiondesignated using the second user interface is matched against matchinginformation in which a matching-use face development image andmatching-use feature region for each person registered in advance areassociated with each other, and a person to serve as a candidate isselected based on a matching result, generating candidate face imagedisplay information for displaying, on the screen of the display device,the matching-use face development image corresponding to the selectedcandidate person together with the feature region.
 18. Thenon-transitory computer-readable recording medium according to claim 14,the program further including instructions that cause the computer tocarry out: generating candidate face three-dimensional image displayinformation for displaying, on the screen of the display device, amatching-use face three-dimensional image corresponding to the selectedcandidate person, based on three-dimensional data of a head of theselected candidate person.